Skip to content

gh-122622: Add PyREPL simple_interact.statement_submitted_hook - #157241

Open
edvilme wants to merge 27 commits into
python:mainfrom
edvilme:pyrepl-hook
Open

edvilme wants to merge 27 commits into
python:mainfrom
edvilme:pyrepl-hook

Conversation

@edvilme

@edvilme edvilme commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #144724

Adds a statement_submitted_hook hook to _pyrepl.simple_interact that gets triggered after the user has submitted a statement (not necessarily a line) but before it is executed.

Rationale

This should be configurable by external tools such as IDEs who may want to perform an action before a statement is executed, such as displaying custom escape sequences for terminal integration, handling custom logging logic, performing analysis, etc.
This hook should allow said tools to perform an action.

Example (Timer)

>>> import _pyrepl
>>> import sys
>>> import time
>>> class PS1:
...     start_time = None
...     def reset_start_time(self):
...         self.start_time = time.perf_counter()
...     def __init__(self):
...         _pyrepl.simple_interact.statement_submitted_hook = lambda x: self.reset_start_time()
...     def __str__(self):
...         end_time = time.perf_counter()
...         time_delta = end_time - self.start_time
...         return f"# Execution took {time_delta}s \n>>> "
...
>>> sys.ps1 = PS1()
>>> 1 + 1 
2
# Execution took 0.0021207999670878053s
>>> [i for i in range(100)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
# Execution took 0.003159299958497286s

Example (VSCode)

VS Code uses custom escape sequences to indicate when a command execution has started and finished. Here, the hook could be used to print an invisible escape sequence to the terminal. (Note that the example uses ### EXEC ### to represent it visually, but escape sequences are invisible)

image

With error reporting

image

AI Usage

AI was used to help with merge conflicts

@brettcannon
brettcannon self-requested a review September 17, 2026 21:15
Comment thread Lib/_pyrepl/__init__.py Resolved
Comment thread Lib/_pyrepl/simple_interact.py Resolved
Comment thread Lib/_pyrepl/simple_interact.py Resolved
Comment thread Lib/_pyrepl/simple_interact.py Resolved
Comment thread Lib/_pyrepl/simple_interact.py Resolved
@bedevere-app

bedevere-app Bot commented Sep 17, 2026

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Comment thread Lib/_pyrepl/simple_interact.py Resolved
@edvilme edvilme changed the title gh-122622: Add PyREPL pre_execution_hook gh-122622: Add PyREPL simple_interact.statement_submitted_hook Sep 19, 2026
@edvilme

edvilme commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again

@bedevere-app

bedevere-app Bot commented Sep 19, 2026

Copy link
Copy Markdown

Thanks for making the requested changes!

@brettcannon: please review the changes made to this pull request.

@bedevere-app
bedevere-app Bot requested a review from brettcannon September 19, 2026 01:20
@brettcannon brettcannon added type-bug An unexpected behavior, bug, or error needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes labels Oct 1, 2026
@brettcannon brettcannon added the topic-repl Related to the interactive shell label Oct 1, 2026
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Oct 13, 2026
edvilme and others added 2 commits October 13, 2026 14:51
Preserve both ANY and MagicMock when merging the interactive console tests, keeping upstream prompt coverage and statement-submitted hook coverage intact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 51fc76fa-b241-49aa-830c-840d2d812b54
@brettcannon

Copy link
Copy Markdown
Member

@pablogsal @ambv feel free to review, but if you're too busy I will just plan to merge in about a week.

@johnslavik johnslavik left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is (primarily) a _pyrepl convenience for VS Code, I'd try to make this PR smaller.

Comment thread Lib/test/test_pyrepl/test_interact.py Resolved
patch.object(simple_interact, "append_history_file"),
patch("_pyrepl.readline._setup"),
contextlib.redirect_stdout(output),
contextlib.redirect_stderr(output),

@johnslavik johnslavik Oct 31, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't rely on mocks of internal attributes.

I'd test this with a PTY instead, maybe in test_pyrepl.TestMain?
That ensures the things that need to be displayed to the user are actually displayed to the user.

Example:

def test_showrefcount(self):
env = os.environ.copy()
env.pop("PYTHON_BASIC_REPL", "")
output, _ = self.run_repl("1\n1+2\nexit()\n", cmdline_args=['-Xshowrefcount'], env=env)
matches = re.findall(r'\[-?\d+ refs, \d+ blocks\]', output)
self.assertEqual(len(matches), 3)
env["PYTHON_BASIC_REPL"] = "1"
output, _ = self.run_repl("1\n1+2\nexit()\n", cmdline_args=['-Xshowrefcount'], env=env)
matches = re.findall(r'\[-?\d+ refs, \d+ blocks\]', output)
self.assertEqual(len(matches), 3)

(This applies to all tests in this PR, I'd try to fit all of them into a few tests in test_pyrepl.TestMain)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A PTY test makes sense here. The unit tests still exercise real statement execution, so I see no need to replace all of them.

One detail: the example needs flush=True, or the execution marker can stay in the buffer until the statement finishes. Our PTY helper uses -u, so it will miss that case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added e28071a0a0a: Add PTY test.

The new TestMain test installs the hook through actual REPL input and checks that labelled hook errors reach the terminal while subsequent statements still execute. I retained the focused unit tests, following Pablo's suggestion. The inline escape-sequence example has also been removed in b7e9986fc46: Simplify comment to remove example; this PTY test does not claim to validate buffering behavior.

Validation: .\PCbuild\amd64\python.exe -m test -v test_pyrepl.test_interact test_pyrepl.test_pyrepl -m '*TestStatementSubmittedHook*' -m '*statement_submitted_hook*' passed on Windows: 9 hook unit tests passed, and 2 PTY test variants were skipped. Unix execution of the new PTY test is still pending.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is still relying on mocking internal attributes, is it possible to not do it that way?

Comment thread Lib/_pyrepl/simple_interact.py Outdated
try:
statement_submitted_hook(statement)
except Exception:
console.showtraceback()

@johnslavik johnslavik Oct 31, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's now easy to misread a traceback from the hook's exception as an exception that comes from running the user code:

>>> from functools import singledispatch
Traceback (most recent call last):
  File "<python-input-7>", line 2, in hook
    1/0
    ~^~
ZeroDivisionError: division by zero
>>>

(I've set the hook in the session, but even if there's internal VSCode paths included in the traceback that are telling this is some hook hackery, I still think this could be improved.)

Additionally, that traceback is displayed before the user code exception traceback.

I'd prefer:

  1. Reporting the full traceback to reader.debug() (a REPL hacker can then uncomment debug() code for verbose error reporting)
  2. Reporting that an "internal" error happened to reader.error() error overlay (something like "exception occurred while running an internal prompt hook")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree the traceback needs a clear hook label, but I prefer to keep it visible. reader.debug() currently does nothing, so that loses useful details.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a behavior issue with console.showtraceback() here. It replaces sys.last_exc and the other sys.last_* attributes. If the hook fails on each submission, pdb.pm() sees the hook error instead of the previous user exception. I think hook error reports need to preserve that state. This also needs a regression test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 866576906ce: Show traceback as a hook failure without modifying sys.last_*.

The full traceback stays visible with an Error in statement_submitted_hook: heading. It uses the existing console renderer directly, preserving coloring without overwriting sys.last_*. Added regression tests for coloring and for preserving both existing and initially absent last-exception attributes.

Comment thread Lib/_pyrepl/simple_interact.py Resolved
Comment thread Lib/_pyrepl/simple_interact.py Resolved
Comment thread Lib/_pyrepl/simple_interact.py Resolved
@edvilme

edvilme commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Hello, when you have the chance, can I please get another review here? @pablogsal @ambv @johnslavik

@johnslavik

Copy link
Copy Markdown
Member

@edvilme Could you please use your own words when replying? Thanks

Comment thread Lib/_pyrepl/simple_interact.py Resolved
Comment thread Lib/_pyrepl/simple_interact.py Outdated
Comment on lines +161 to +162
console.write("Error in statement_submitted_hook:\n")
console._excepthook(type(exc), exc, exc.__traceback__)

@johnslavik johnslavik Nov 17, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_excepthook writes to console directly, but it would make sense to me if we reused sys.excepthook like _showtraceback, just without writing to sys attributes. Maybe add a param to showtraceback and _showtraceback for REPL-specific errors to just avoid writing to sys attributes but go through the traceback display pipeline inside _showtraceback? Or simply move the pipeline to _excepthook, since no call sites (other than _showtraceback) use _excepthook directly

WDYT @pablogsal?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting merge needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes topic-repl Related to the interactive shell type-bug An unexpected behavior, bug, or error type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants